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ABSTPACT 

Certain^ features o£ the PLATO i? systec allov tb« # 
gathering of large quantities and varieties cf infotiation describing 
the interaction of students with coiputer based education courseware. 
Twoifcinds of inforiatioc can be provided: (1) data concerning single 
student interaction, which lust be analyzed ianuallyf and (2) data 
su»«ari2ing activities of a student, after lany* interaction s, which 
can be^ lanipulated by the PLATO system. Ontil recently, this type aiid 
voluie cf d$ta have not been^asily available.* Thisr report describes 
soie initial atteapts and the current status of efforts to collect, 
condense, and analyze these data for the 'purpose cf diagnosing 
student problems and iifroving instruction. (Author) . 
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AjJ^novledgments ^ • ^ • , . ^ . 

'The jdatra handling and analysis rouWnes described .in-" 
•■''this report have played an iisport^nt Dart in the development 
^f, the PLATO. IV courseware. In addition tc the use of these 
r6ut44|gs by tire tTTC/PEER gr^up^in the^r evaluation of the. 
A'RPA'/PLATb sites', many other /mtlitar/^nd civilian ^it.es 
have made long and profitable use of the data collection arid 
analysis packages available on the PLATo system, ^uch of 
the information in this report tias been gleaned from studies 
conducted as part of the CERL evaluation of the Community 
ColLp^ I^roject. In addition, Ediftrational Te.stljig Service 
(E3c§), the independent evaluators . for that projgjtt contri- 
buted suggestions and findings when they encountered prob* 
' I'ems with/their own data collectiojv programs, and elected to 
use these routines* Finally, a large buf unrecorded . number 
of inxjividual PLaTO' curriculum developers have employed 
t^ese well-publicized lessons as part of their develO;pment 
process and have provided feedback to help the creators 
refine the routines. ^ 

The- authors' would like to ackriowledge the contributions 
of -^aany people toward the advancement of the techniques for 
collectj^on and interpretation of area summary data. The 
routines were prepared over several years by R.A. Avner, 
Kumi Tatsuoka, and Tasar Weaver aided by Elaine Axner, 
Steven ^ Boggs , and Hartin . Siegel . The initial users of the 
routines provided many helpful sugg'estions and aided 
ir\terpretation ^of the data. Thos^ users were: Lou DiBello, 
Barbara Lederman, Keith Bailey, Rary Kanteufel, and Frarfce? 
Kennedy. Data editor and missing data correlation routines 
were prepared by*Hichael Feltyy^ multiple regression program^ 
by ^Cikumi . Tatsuoka , and . transformation programs* by Tamar 
'Weaver. Recent investigations into^^the use. of time data 

have been carried oat by Kikumi and Maurice Tatsuoka, This 
'^report was prepared with the aid ^of A. Lynn Hisselt; the 
^ manuscript was edi'ted by Elaine Avner, typed by Julie 
.Garrard and illustrated by 'Roy Lipst:hutz. 
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AiJ^novledgments ^ • ^ • . . ^ . 

'The datra handling and analysis rouWnes described .in-^" 

•-^this report have played an important Dart in the development 
-of, the PLATO. IV courseware. In addition tc the use of these 
r6ut44Bfs by tire ffTC/'PEER gr^up^in the^^r evaluation of the. 
A'RPAVPLATb sites', many other /mtlitar/^nd civil ian ^i tes 
have made long and profitable use of the data collection arid 
analysis ' packages available on the PLATO system. *^uch of 
the information in this report 'fias been gleaned from studies 
conducted as part of the CERL evaluation of the Community 
Coll^gt I^roject. 'In addition, Edi/trat ional Te.stl^g Service 
(EI§), the independent evaluators . for that project contri- 
buted suggestions and findings when they encountered prob- 

• Terns with/their own data collectiojv programs, and elected to 
use these routines* Finally, a large buf unrecorded , number 
of individual PLATO' curriculum developers have employed 
t-hese well-publicized lessons as part of their development 
process and have provided feedback to help the creators 
refine the routines, f 

The- authors' would like to ack-nowledge the contributions 
of -isany people toward the advancement of the techniques for 
collectj^on and interpretation of area summary data. The 
routines were prepared over several years by R.A, Avner, 
Kumi Tatsuoka, and Taear Weaver aided by Elaine Axner, 
Steven ^ Boggs , and Har t in . Sie^el . The initial users of the 
routines provided many helpful sugg'estions and aided 
irvterpretation ^of the data. Those users were: Lou DiBello, 
Barbara Lederman, Keith Bailey, Hary Hanteufel, and Frarfce3 
Kennedy* Data editor and missing data correlation routines 
were prepared by*Michael Felty>^ multiple regres^sion program^ 
by'Kikumi Tatsuoka, and . transformation programs by Tamar 

'weaver* Recent investigations into^the use, of time data- 
have been carried oat by Kikumi and Maurice Tatsuoka. • This 

'^report was prepared with the aid "of A. Lynn Hiss'elt; the 
mafiuscript was edited by Elaine Avner, typed by Julie 

, Garrard and illustrated by *{loy Lipslihutz. 
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At>stract 

" • * 

'Features, of the/ PLATO* LV CBE sy^.em allow 'the* gather i^g 
of large quantiJtii^s and varieties of Information describing 
the interactions . of students with CBE courseware. Until 
recently this type and volume of data have not been Easily 
available, and hence few techniques or guidelines for its 
analysis have been investigated. This report describes 30ffle 
ini-tial attempts and the'^urrent status of efforts to 
collect, condense, and analyze these^data for the purpose' of 
diagnosing student problens and , improving instruction. 



Prologue • . t ^ ji / * . ' 

* * ' k » 

- ' » " ^ * *■ 

• . • . . • ^ . . . 

^ Throughout the history of the PLATO system^ directi^^-ns 

in data collection and- analysis-^ha ve shifted steadily. // 

, Shortly after the -development of early PLATO systems, "^a" 
system designed to ^roci^s. detailed individual data, the 
System for Instructional Response Analysis (SIRA) was 
devi'sed (Avn^r, 1969). This packsge could analyze •student • 
responses vi;-tually key*tfoke-by-keystroke • It was designed 
to perform many, of the highly sophisticated analyses made* 
feasibr«uby ttie power of the computer. Unfortunately, it6 ^ 
extensive flexibility overwhelmed most potential users.*'; 
These users, under ^t^zz\st^ to produce large quantities of * 
instructional material could rarely spare the'tfme needed 
to master*th? advaYiced* easurement -concepts* upon which^ the 
SIRA package was based^ Hence, the'y tended tQ avc^lLd- any- 
approach to data analysis that they did not already understand 

haced with a technically sound package - which was not 
reaching its intended audience, the data collection philo- ; 
sophy of systems •designers changed. In order to incre^ase- 
the likelyhood that instructional designers would mak-e fUll' 
use of the stucfent dySta that CSE gav>e access to,, it w^s 
decided to limit sysbe^i-supported analysis -patskages" to very 
basic levels |which. would be easy to use and "hook" users on 
the utility clf^tUdent dat^* it was expected' that once 
tJSers became ^^customed to using ^udenlf data, they would* 

•desire access|to more advanced techniques'. Father than 
attempt to prfrvide the very flexible jadvanced packages tihat, 
would b*e required to support the range of needed^ appl icat4ons> 
authors were to be given tools (commands, 'databases, et^c.) 
which they -would neejt^^ develop their own individual 
analysis packefges. ^^stem'designers felt that specialized, 
packages .for- $|fch prpject, disciplln.e, or group would mean - 



greater efficiency and higher acceptance than a "super- 

package" "for all users* '^hus, oaly basic routines have 

been systenj-suiiported since the advent of the, PLATO. IV system 

(Tatsuoka 4 Siegel, 197^). ' 

At ftrstj aS anticipated,' varioua groups began developing 

thlfeir own ' analysis .routines (Smith '/c Sh^rwo^d, 1976; Yeager,*. 

1976; Weaver, 1975). ySbwever, the next step of the process 

did not occur exactly \as expected. Rather than each gr-oup 

developing^'tinigue packages tailored to their. needs, some 

groups began 'adopting or adapting exisjti^ig -packages Writteri 

by others. The resulting p-ackages were Upgraded, and consali- 

dated until toda^ there again exists a set of multi-usec * 

students response analysis routines. ^ (Of course, many groups 

still naintaih thei^r own routines,) The current general- 

purpose product is somewhat different than the SIRA product: 

fOl^ example,^- it does not not ydncltrfe'^many of tTie detailed 

fui^tions possible with the SIRA package. * Happily, the 

curpent package does receive broader usage than tHe SIRA 

pacfKage. , ^ " 

J Thus the evolutionary process has come not quite full 

circle. Though the autho>s of this^r^port caYi venture no 

opinion concerning the treijdsi'for future a>ialysis packages 

in terms of user-produced or syisteft^-produced programs, new 

syste'm-level data collect ion ' command^ and continuing impcove^ 

ments to the varioAJS user-level analysis packages suggest 

that, further changes in th'is area 'are probable/'- The follow- 

ing reporf'details the^ creation, development, and use of a* 

set of , 'student data response routines collectively knowh as 

"the -area— ~p36kage." . The phase of development which is 

« 

described herein coincides with the implementation of th'e 
.PLATO IV system. • - ' . " ' ' ' 
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Introduction"^ 

One of tne virtues of automated instruction expressed 
^by CBf advocates is that it allows- easy Snd unobtcusive 
'gathering of vast arabunts of data about a- student'V. 
p^rfoMnan6e in ^.lesson. All this^ information c^n be used . 
to immediately analyze, and .impVove^the ii^ tract iprial 'product, 
Nevertheless, it became -obvious in the later years of the' 
^PLATO IJI system and the early years of the PLATO IV. system 
efficteot studeAt da^ta collection package alone was n6t 
sufficient to^fully exploit the information availa'b^e. The 
full potential of CBE could be realized only by means of 
computer-aide'd analyser/ of th^ student 4^ka. 

V ' ^ I ^^^^ V 

, $s the PLATO^lV d^ta collection system was berng 
created,' the gathering two kinds of information »was th^s 
provided for. The first .type of dS^ta deals only wij;h a/ 
sin^e student interaction: the response he made, a -key he 
pressed, or an error he caused* - The second type of' dat\ 
suTnmari^es the activities of a student after many interactions 
for example, rather than recording responses, it categorizes^, 
t^hem and tallies the number correct and incorrect. ^The 
first 'Kind of data must be analyzed manually by the author; 
the second can be manipulated by the -PLATO system, then 
interpreted by the author. • ' * * • ' ^ 

As part ^of the software for Collecting* the summarizing-' 
tyj^ .student datay the PLATO IV system ^provides the author 
with the ability to divide a lesson into*contiguous "areas'^ 
and ,to automati'cally record a variety of measures about the 
student's use of the lesson-- within each area. Because this 
provision is so completely automa.ted , many authors have 
implemented this feature,' and there is* a great deal of so- 
called "area data" bein^ collected on the PLATO IV system. , 

An 'area' is an author-designated segment\of a lesson 
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^tiich fai^y consistently uses *'a single strategy "I'ike drill, 

tutorial, in'c^uify , fete, to *teach ^ related iset of facts or 

y concepts. It is typically about 3' to 20 minutes in length. 

Whenever a student completes an- area' or stops \/orking on a ^ 

lesson,^ the computer records .the following information for ^ 

that student:. tl\e time spent in the "area, the number off 

questions attempted, the number of questions eventually ' ^ 

answered correctly, the number of incorrect student responses 

which^were anticipated anjd'' unanticipated by the author, the<. 

'number of questions answer ed 'correctly /On the- f.irst try,'^, 

counts of ins tanoes • in which the student requested and 

received on-line help (and*cases when dn-line *help .was t 

requested but unavailable), and the n^rSe of the student^' the 

J.esson, and the area. This " * are*a^summary, ''as well ag the 

individual * interaction data (e\g.,. the student respons§s) is 

stored on-line for later sorting, transfer, or printinij. ' 

It should be apparent that it is not only possible, ^ but, 

quite eas5^, to gather appropriate st^udent data- to *be u^d to^ 

revise Tessons and monitor studjsnt' progress-. The remaining' 

problem'^^ that although tJfie onde-dif-f icult ta^k of d.ata ^ 

collection has been solved, the analysis of the resultant 

volumes of information becomes time-conSujning and f rusj^rating, 

Recording all student responses (typicallv one^ pef minute) 

for a clajs of 30 is cl^rly out of the quelstionV Individual 

student interaction data must be recorded very selectively. 

Even- the area summary data collects ratber quickly. For - 

example, ^ iilass of 2-10 students tsking a two-hour lesson 

which provides area d^ta about every 10 minutes vill 

generate 2500 area summaries, each containing more than 10 

items oC, hiatal There are so many 'values *to examine and po 
• » ' * 

many comparisons tjhat might be made that many lesson * 
develppers, unpble to analyze all their data, analyze none, 
of, it. - « • ^ 



) 

ERIC 



10 . 



The' analysis of this' massiv^ amount of dati .demanctg t/ie-" 
use* of computer. Befor^ the "use of large CBE systems- such 
as PLATO IV, the few .CBE. users in existence had .comparative'ly 

^■Uttle courseware for which to gat-irgr ' data and little soffc- 
are to ma1<e data collection feasible.' Thus there was a 

.i^imited need for ways to sort', condense, and analjrz.e tfie • - 
data. For th^se reasons, PLATO lesson developers who gather 

>rea summary data have had few guidelines^r its interpre't- • 
ation, .The rest of this report descr ibe/ recent attempts 
and the current status of efforts to dondense, analyze, and' . 
interpret area summary data for the*J>ijrp*ose of improving 
in-strqction and diagnosing student problems. . * 

Interpretation of Area Summary Data 

* Two measures of .student interaction have been found .to 
be especially valuable for ev,aluating the perfor,mance of " - 
students and lessons: (a) the percentage^,of guestion* . 
correctly answered" on the first attempt (abbreviated 
Ibkfirst), and (b) the number of incorrect, responses 

question (abbreviated errors/question or. errors/ques) . 
Instructors comparing 'ci^fisroom 'observations .to values for' ' 
the above Rarameters find, consistent , intuitive conformance. 
For example, an area. which is very difficult will require 
the classroom proctor to aid th-e students and will result in 
Sr low^^okf irst and a high value for, errors/question. ' 

For a wide .variety o.f subjects, and disciplines a ala'ss' 
avera|e of 80 Jokfirst has been found to indicate that the. 
difficulty of the material is wejl matched to the class. 
Instruction with Jokfirst lower than 75 is generally too hard, 
and material with ^okfirpt above 85 is typte^lly too' easy. . • 

- The Jokfirst alone is not sufficient. When only a few 
of the questions.^ahe very hard and the 'rest' .are -easy , 



%x>kfirst has vits ' usual--valire/--Qnly errors7question if^ ^ 
.Sensitive. Conversely, ernons/questlon is., not &s sens^ive 
'as Jokfirpt "in cases where the dlfffcUlty is .qpnsisterttly 
'Slightly too hard, but is divid^ed evenly between ."questions * 
In some cases^ one, finds that reasonable ex<5lanat ions cran be 
^-matie for "unusual" ratios, but in most instances, legitimate 
^problems are fotind, • * * ^ 

Despit'e -large student-to-sttid,ent Variability, ^erages 
of Jokfirst"^ and errors/qwestioh (and completion time) tend 
to be fairly ^stable* It*'appears th^t if ^.essons were 
' accorrtpanied by a 'set of normal^ r9nges for various st'judent 
populations, an evaluator or instructor could estimate -the 
aptitude and/or preparedness *of a class. However, it -Ss not 
generally feasible to use %6J^f irst or -errocs/quf s^t jfon bo * 
•'il'grade" individual students./ ThiV is becai^e extreme values 
- * for certain studetnts ma/ poiAt out cases which need further 
investigation, but , variationV^ fn the s^^yle-of student usage 
often explain what initially appear to be "probl^^^." For 
example, some students deliberately make errors to see vhy . 
alterna.tive plausible arlswers^are incorrect. .Currently, 'the 
greatest g^enerallzabl^ pse 'Of area summary dat^ JLs f&r' 
guidance in the revision of ]^essp*ns. • - \ ^ • 



An Example of How Area Da6a Aided ReVx^iori'jOf. a Lesson 

Irfitial yersion -.\ A irtath les.son teaching the addition 
and subtraction of^sigwed nuinbers .was divided into seven 

\ ar^'asp Each of^the firsti six areas' ^contained instructions,. 
exampieQ*', ^and 4paercLse4 d.ealing witj^ one type*of problem 
(e.g., adbing two numbers 'of qpppsite sign, or subtracting' 

• two numbers both of which are negative). The 'Seventh ^gred . 
provided mi,xed- practice^ with the previous' six problem types; 
Dati for several classes is/foi/nd in' Tabl^ 1. Thtf^ lesbbn - 
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Class Perfotmarice Data fbr the Original Version of' a Math 
Lessoti ' \ ' I '' ' ' ' - ' 



wias. first used by large 'numbers of 'students in Spring ' 1975 . \ 
As Smicipated' by the authors,. 'the difficulty 'of 'thfe fittii 
aad sixth, areas *(subtraQt).ng* witft. negatiyes) was greater 
than that of Qther areas* However what was' not expected 
was the very great dif ficulty/of the final* m'ixed exercise. - 
In fiv^ of the six classes the ^okfirst dropped off 
dramatically for. the for the mixed ex'ercise (the seventh, 
area). 

Tbese^data suggested a need^tp add an additional "help'' 
area before mixing together all six types of problems.' 
Because individual problem types were apparently mastered, 
discrimination between problem types -was seen as the' 
stumbling* block . ^ ' ^ 

' At) exg^mination of the histograms of Jokfirst for each 
ar.^a provided additional insight about what was l^ppening to 
the students. Figure 1 presents the rejsults of a represents- 
tive class for the last three areas'^o'f the les^^on. These- ^ , 
and^other 'data suggest tna^ many students appar'eO'tly did 
weir throughout the lesson 'whereas. part bf the class prum- 
•'^meted rathei^ suddenly on the last mixed exercise. * T^is 
information, combined with analyses- of^ndtvldual-^stu^fe^ts, 
led the author to decide that only students, who scored lower- 
than' 80%. on the previous areas would be routed through the. 
adrfitiohal disqrimination exercise ,.Ci •e. , '€hje "help" area). \ 

A*second, simultaneous modlif ictfiorv tp thi's lesson was 
based on instructon requests -rather th^n student data or 
Comments. 'In ordfer to ireduce the time needed f,9r the lesson 

the numbel" of example^ in e&ch ar^ea- was reduced from two to 

' -\ ' "... ^ ^ ^ 

one.. • * ' ^ . ' ^ ' . 

^econd version : The results of Che Spring and Fall 

classes for the first hand second vef siorts of ] the lesson 'are 

showa in Figure -2. Tb^ .classes using'the revised* version* 

had a 'ufiifdrmly lower performance las measured by iDkfirst). 
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Figure 1 * • **' 
c Class -'Perf^mance Qj/pinal Three Arear's^of a Math Lesson 



Although the consistency of . the shift /In' performance might 
possibly have been '.caused by FallMgT^ students who -were 
^ more Intelligent than the Spring 197^5 studerit;5,' the- re<luction 
in the number of examples was stroilgly implicated as the 
culp,rit. IQ any case^ the less 

ance suggested that it be 'cjodlf l^d , no patter what the cause 
might be. Therefdre, one or, mo^e examcTles we,re* added back ^ 

- into air teaching'areas except area 4/ in which students had 
scored higher than 80$ okf^ f/r the revised* lesson 'Btifdent- 
byrstudent Analyses indic^t/d the-"lieLp« section seemed 'to 
-ber aiding poor students fis w^ intended, but the ^change in - 
overall per*fo'rmance causetl by the^ reduction in number of 
exai^ples was so large- that Curther irrterpretation ^as 

* obscured. ^ . - - 

Thlrd^ version . Data for the .t)ilrd* version was co^ected . 
in itfll 197^. The data from the'first two versions w* 

- collected from the sdme coamyrtity college classes, whereas 
the data from the third version w§s ^thered from iunibr ' - 
high 'school studints.^, Heverthel^ss^ the results of the 
analysis Suggest* the' lesson behaved /similarly for all three 
groups of students (Figure 2). 

Th'fe examples addetl bo the third version seem to 'have . 
raised the/'Jokfirst- for all areas except' area-fl.^ The Tact 
thft ,the /humber cif examples In^rea 9 was not jchanged con- 
tirms tKe validity of the - chaijgesr to other areas. It is 
interesting to Jiote >tha't although the Xokfirst for .this 
■ last/version is genef-ally Ipwer than that of iihe first versiori,'^ 
th/ performance on the final- miVed test is higher I- Thi^ is. 
Probably due to the' "help" ^rea^ which was not available to 
Students taking the origijial form of the' lesson. The Jokfir^t 
for the help area is lower than ^11 ^her areas because .only 
Students doing poorly in previous sections are routed thrdugh . 
the help area. This extra help and practicife in discrimination 
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of probl6q types tends to keep .the poorer students froitf 
^falling apart" on the final mixed exercise. The result £s 
that the a^verage sFo^e for the whole class, rather 'than 
slowly declining through the later areas, makes a sharp ? ^ 
upward turn in the final test area*. The class average' of 
tokfirst on the f^nal test area is near the average Jokfirst 
of the previous six areas* TKis is* where one would expect 
it to be if Vfve'^diff iculty with discrimination of problem 
types 'had been eli^ninated; * ^e^test samples items from each 
of th^e previous area^» . '* 

Looking at.t^e individual student performances (not 
shown here),' one sees that, 'students whoodid. very well init- • 
ialXy and who thus avoided l^he ^'help** area receiveif^^a Iqw^r * 
score on the miUed practice' than fehe^r average- or the six 
problem types. y Conversely , student^ that did poorly and* got' 
the review, did betted on the mixed practice than their 
previous average. This s-uggests that a ,sec^d briefer help * 
sectrion to teach "good** students to di'scrlminat6*^roblem types, 
would be^ an ef f icie/it ^ajr to further increase the average 
score. ' * \ ^ ^ *^ , * ' 

Summary . While lesson revision sJ}o(jld continue to be 
partially based on* comment^ from students anci instructors as 
well as ^uihor intuition, the preceding example demonstrates 
the power o'f basing revisions on student performance data. * 
For example, the decision to re-introduce examples can be 
explained and justified to skeptical iristructors by means of . 
, these data.* a decision wereoCsade that the student scores 
should be increased still further, the absence of data such 
as that shown here would" make it unclear whether more exa%*^ 
Jte^ or more practice should *e added g^nliich types of 
Questions need' the most emphasis Jji all likelihood, 
increasing either examples or^ practice would i'licrease finaT 
scores, btjt only by Substantially tincreas4,ng lesson -cbrop- 
letl^on time^ As^, noted above, there is evidence/ that a * 
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second brief help section for more of the students might 
substantially; boost average scores. If so, that would be' a 
very efficient revision in terms of both 'the^ students * and 
the Author's tise. 



*^easures of CBE Interaction 



f 



As was previously .noted,* area summaries provided by the 
conouter for each student cojitain tallies of various of the 
Student's activities. Jn \ife lesscyi revision, efxample , 
however, none '-of the data was used in fts raw form — instead 
^^a ratio or percentage was calculated. Because the number of 
questions varies from area to area, th'? number of questions 
answered correctly on the first attempt cannot be comp-ared 
fros one ar?a to another. However, the ratio of ^qu^stions V 
answered" correc^tly on first try" to "total questions" can be 
*^ed to coas'tTuc^ a parameter which can be usefully ^mpared. 
Kor convenience, the ratio is often expressed as 3 percyen^-"' 
age. > / ^ ^ ^ 

The lesson revision example was based on the exataination- 
of ,a single parameter 'derived from area summary data. In 
fact, there age many more data available and sj^ny more ^ 
paraneters whic7> can be derived. Potentially useful ratios 
(in additidn*to thdse already mentioned) Include: 
qyestions/minute (a messure of the student's speed and/or • 
the nature of the lesson), interactions/minute (where 
ijiteractions is the sum of response* attempts/,- pn-line help 
, accesses,' and students-requested branches), and ynanticipated 
responses/question (a'measure of Ifesson "polish*^). 

The reader can imagine' many oth^r measures whicTi can be 
e9nstrupted by algebraically man-ipulating the' parameters / 
previously lifted. 'At this time however, only a fev ratios 
like Jpkfir%t and" errors/questitJn have been investigated. / 
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For the^ other ratios, little is known about the validity,. 
generafliz.ability , or normal range or values. 

• . /• . . r . - 

Features an<i. Options for Data Analysis 

The area analysis papkage automatically calculates many 
useful ratios and averages as well as allowing the user to 
const^HJCt iiis own. 'Other dat^ can also be retrieved, mani- 
pulated, and displayed" in a vast vafiet/ of formats by the 
data analysis routines. Rather than^ being exhaustive, thi^ , 
section lists examples sone of the especially useful or c 
interesting options. 

t V 

To aid the user in finding relationships among the 
student datd, ' scatterpldts ^nd histograms may be-formed. 
After a histogram has-been displayed , it is ofj:eri interest- 
ing to investigate the [ characteristics of data at the ends 
of the distribution. Therefore, one option allows retrieval 
of complete information for all records of a type specified 
by the user (e.g., all time data for any student who com- 
pleted a specific area with three or fewer errors). 

\ Tables o^f summary data are provided in three categories 
all areas within a Igsson, all students within an area, and 
all areas conpleted by one stjudent. Each summary^as a* ^ * 
•different focus, a^d inconsistencies in summary data may* 
point to potential problems in areas-, lessons, or students. 
An especially interesting visual displ^ shows tMe progress, 
of an individual student ral^rtXy&sto the rest of his class- 
mates (seei^iguire 3). An 4pproprirate parameter '(okf, time, 
etc.) is plotted for up to 18 consecutive areafs. Th^ mean ^ 
and standard deviation ' for the'^class are visually indicated 
and the student's score is. plotted over this "background 

.^A student may complete ^n area several times if the results^ 
from his first attempt are unsatisfactory acpording to his 

. • . • / 
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own or the i^ss'on*s critef-ia. The .results of each success- 
ive qoDipletipn are indicated so that the ;stud^nt's improve- 
ment can b/ verified b*y ihe author ' or by the sfcUdent*3 ^ . 

nstructori . It was this kind of display thati was us^d in 
the lesson nevision example to follow the success of indivi- 
^duai_"g^j^d" and "weak" students to see how the lesson -was^ ^ " 
p^fprming. ' ^ * ' . • " 

Another technique on6 may use to concisely portray /* 
Targe quantitid^s o£ data visually is to generate S cumula- 
tiye' fre.qufency curve (called ^ ogive)'^for the^ompletip^n, time 
of the areas (Fig-ure ^) . Varying^ the scale, factors reduces 
the data from different ar^as inta a familiar S-shaped cur\^e, 
whose shape and parameters may tben We interpretted, (See 
discussion under "time" in the next se6tioh.) 

"^^^ny data ft^om an area summary may be tran^Xerred to a 
more powe^gul set.^of statistical analysis routines available 
on, the PLATO system/ After transfer, the data may be trans- 
formed by an arithmetical function (e.g. reciprocal, loga- 
rithmic) and used in multiple regression routines. Other 
st^i/dent data-not found in' area summaries inay be/included in 
the regression (e.'g., grade point average, age). ^ 

The graphical or tabular displays -produced by the 
analysis pacJ<age§ can be iiardcopied by^means of a plasma 
display copying dev.ice (e.g., the Varian copier)^. Iri 
addition, most of the tables, of data can be automatically _^ 
formatted for a line printer. • • . 

Analysis of Military Data • - . . 



The data analysis routines were rgfined and expanded as 
part of the ARPA/PLAT<? 'evaluation project in anticipation of 
their use during formative and summati^^e evaluation ^period s.^ 
At the inception'of the evaluation phase of ^his project^ 
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' .. . 
the lessons at, Aberdeen >roving Ground had been compietely 
wri"tWn— a»<l-4;ested ; students wpre no Jonger using them.! At 
Chanute AFB, all Wessons had been written and validated just 
prior to the start <xY tiie evaluation period, but student use 
,was>continuing. The HTC/PEER evaluators, being jespecJlally^ 
interested in- studying and aiding th'e process of lesson 
devy^lopment an.d revision, thus ^directed- their efforts tpward 
^he lessons of another major ARPA/PLATO §ite, Shepparcl AFP. 
The Shepp^rd project had beguft a year later thSn other , 
two projects~~an^^TSrB^ui4c^-Qi^^ lesson development 

and validation was slated to f all into^Tlie evaluation 
period. Unfortunately , the small class size (16 of^fewer 
for each of two xrlasses)'and the evJJjtual re-direction of 
that project produc^d^data that ^as sufficient for only a* 
few analyses. 

foreseeing som^of these problems , the MTC/PEER evalu- 
staff elected to collect data from Chanute AFB students. 
Because ^e Chanute staff^d4.d not ase firea^ summary^ informa- 
■^tion durin'^validation , no provision had been made Tor its 
collection, therefore MTC/PEER st^a^. reprogramn^d eight' 
lessons we had previously reviewed, d^^iding eacrK lesson 
into areas and installing other programrai>rg required for 
collection of area data. • " ^ " 

^ Because we were forced'to rely on the forbearance of 
Clfanute staff for a number oj (J ata-han'dling procedures^ 
unrelated to their oWcO- goals, data collejction could be \ 
maintained for only a limited period of time. 

Chanute data . Because the Chanute AFB lessons had ' ^ 
already met their validation criteria, major revisions ,bas,ed 
on *thB arJa summary data appeared. unlikely . Therefore the, 
analyses {Sarformed with these data were directed toward 
ieveloping ^general hypotheses about how to Use afea 4ata 
ather* than toward diagnosing student or lesson problems. 



The exister\ce of large quantities of criterion-referenced . 
test^data, course test data, ittitude surveys, and area 
summary data for 'Chanute students +ias provided a ri<^h source 
for CBE investigation. Current research efforts make use of 
routines which extract relevant parameters from' area data 
and allow them to be compared to a selection of predictors 
Such as those suggested previously. 

^> Sheppard da^ . We t^ansfeVred and analyzed all data 
coil ected^ f otVb^;tfi classes of Physician Assistant* students--' 
summar ies "fDr^56 areas from 62 lessons were obtained. In ' 
several cases there w^re sufficient data to serve as a basis 
for revision of lessorts5; however, the modified goals, of- the 
• projecl plus^a delay in the availatrility of the analys'es 
meant i^at no revisions were actually m^ade on the basis of, 
this datc^^ A post hoc analysis comparing^ the student per- 
formance o^ the first and se*€iond classes ^monstrates ^at 
the student^ population shifted or tlfat revisions to the 
Wessons we/e largely ' successful . For example in 12 lessons 
wheiifi_fii:>fliparisons were possible, both the Jokflrst .and the 

Tors/question improved dr^maticarlly in 10 cases and 
^worsened slightly in, 2 cases. As ^xplained previously,, 
^further interpretations and analysis were not ^made because 
of tTie uncertain future for the^ lessons. Nevertheless', vith 
the potential f6r efficient revi^ionr and valTB^t'^on demon- 
strated in this init-ial .^ffort \jj MTC/PEER sta^, the 
Sheppard PI^O gjoup has now^ta/:^n over the responsibility 
f^i^^atherifig area data, 'analy?(itig it, and using it for 
fonmative development, 

/// 

Preliminary Findings 
? \ . 

Time . Initially, user^yof the area analysis package 
concl^uded that time is not 'a^e rally an indication of student 
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success. As an example they' cited' two students (Gerny ''^nd 

Lenerts in Table 2) who SfJent more time^ did many more 

exercises tlTan other Student, and performed well. All 

students had the option to get more practice. Jhese two . 

chosfe to do so and apparently enjoyed it. Hbwever, .^^dent 

Adam took, about as much time as Gerfiy and Lenerts,. but/ did, 

very poorly. Thus the elapsed time in an instructior/al area 

is seen to. be a tesfi-reliable indicator of success ^an 

%okfirsf or .error s/question . ^Other data-, not sh.o>fri here, 

indicate that in "testing" areas, as opposed to teaching or* 

prac^tice areas, fast students \end to p'erTorm better than 

3tudents who take a lot of tjme to complete the test. 

Investigations of cumulative frequency curves fqr time 

data (Tatsuoka & Tatsuoka, 1977) promise to' supercede the *' 

early conclusion about the usefulness of^ time data cited 

—'above. The new research attempts to relate tine and 

* 

performance "data by fitting, student "data to a mathematical 
model and equation. There is an indication that data -which 
'does not fit the model well may point oUt problems with a 
lesson segment or^ studenfc. population. For example, ♦Chanute 
AFB data from a lesson whose data fit the model especially 
poorly was found to have at 37% failure rate (vs.' the 10$ or 
lower failure required for validation). Students ^taking a . 
statistics lesson (originally written at ,an ARPA site) were 
divided into two* groups based .on their ptsr'formance . The 

time 'data frdm the group with higher performance fit the 

* >- 

theoretical distribution significantly better "than tiiat for* 
the poorer grouj^. * ^ " , ^'^ 

An- attempt is being* made ,to use time data tp provide . 
additional information when choosing whether to pass or .fail 
students whose score is hear' the cutoff point of ^criterion- 
/eferenced test . If one, uses only the student's score ^. 
measurement errors inherent in testing mean that the decision 
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Student Parameters for an Area of -a Math Lesson 
(Note: student names are fictitious.) ', 




to pass or -fail some* students near the cutoff point will- be 
madV incQrr^ctly . ^^ss/fail decisions mSy be made more 
accurately by. Including, additional data aboufe. the time 
nee(|ed by the atudejit to complete the lesson'^ and/or the 
lesson test*. Final reports of the"* research, B4ing support^ed 
by th^ Basic Skill Group the National Institute of ' 
E^faucatioir, will be^avatlable \ate in 1977. 

,. Lesson' strategies . Jf a lesson is^^ivided into areas, 
so that some of the -areas C9ntain mostly didaflic^ tutoria'l 
material with^few opportunities to practice new skills while 
other areas consist primarily of practice with few-help^'* 
sequences, one. will find a sharp contract in *the patterns of 
%okfirst and. errors/question Mstograms.' Figur>es 5 and 6 
exemplify these differences. The di^stribution of both 
Jokfirst and errors/question is broader for the practice 
area than for the tutorial area. , I 

Lessons that teach by "trial and error" method, or - 
lessons that plainly, teach* by help sequences have still 
different patterns of Xok'first. As cpuld- be predicted, 
nqiiiry-type teaching produces a higher error rate. 

Use of incomplete ar^a records . During the analysis of 
area data one , must decide whether or not to include records 
**pf ^^rea data in which the studen*^ failed to complete .the- . 

whole area^.^ Available information suggests excluding such 
!incor^le>e area records when searching for lesson problems, 
.^^his Is* coNjnter-intuitlve: ^one would seemingly want to find 
cases vhere , students were unable to finish an area because 
of ^problems. However, 'the data from* incomplete areaf^ is* 
lively .to be skewed so badly by short-ti<ne users that 1 . 
problems may be hiddea, rather than elucidated, by including 
this data. Vfheh sufficient numbers of completed %*eas are 
not available, data from incomplete areas 'can be "cieaned"' 
by discarding any students who have answered no questions in 
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Comparison of Jokfirst in Tutorial- and 
Practice-type Areas* 
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Compari?oVi of Errors-per-Q'uestion in- Tutorial- and 
Practice-type Areas * ' . 
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an area or who have' spent only a short .time in the area. 
Xiie following' histogram (Figure 7) contrasts data collected ^ 
from "complete onLy" and »*3ll" .area summary Records* The 
shift to the righi for the "complete pnly" uses is visually 
apparent and demonstrates typical diTferences observed when 
trying to interpret . incomplete area records. 

When calculating \e%sihing efficiency or* when viewing 
-the prog-ress of an individual'student , all area data sftouid 
t/e ex'amined so -that both successful' and unsuccessful' 
endeayors are 'prof i led . ^ , ' 

FCiture • , • . 

At least three corfcurrent efforts are needed to advanck 
this field of computer anaiysi;S of student interact ion data: 
' n.. New^ meaningful measures musl be l^ound and validated 

2. Ways to use these measures for the formative dev^l- 
opment of le^sson-s and for project management mu^t 
be found.. * ^ ' ' * 1 . " " 

3. Computer software packages to simplify the analysis 
task for all a*uthors must be built, tested, and 

• documented. * * N 

TJiere are dozens of measures and parameters which'seem 
meaningful; and potentially 'hundreds of correlat ions, that * 
mi-ght be examined. The' exploration of normal valye3 for 
these, parameters and correlations has, just begun. -Intuition 
has ser^LasLas the only, guide until recently — now careful, 
systematic investigation's are underway^ A desireable result 
might be a set of algorithms for reducing the great volume 
Qf area data into a handful of consistent, sensitive 
measures that relate to the major characteristic-^ of a ^ * 
lessen (eVf ectiveness, efficiency, difficulty, etc.-) Enough 
exploratory work has been d^e ^to demonstrate both immediate 
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applications and the potential for the 'future. { 

Current research at CERL inviolves regression analysis 
of the areaydata t?o predict ^lesson and 'course »ores, and 
the analysis of elapsed time for t^^s^xi% end Resting t6 
predict student success. * Other users of area ie%a are ^ 
investigating the shape (i.e., mean, standard deviation, ' 
.sJ<ewness) of distributions for 'parameters sucW as Jokfirst 
to diagnose lesson weaknesses (Smith, Gtiesquiere , & Avner,, 
197^1). '\ ' , ' 

A new manual describing the use of trfe aata transfer 
and 'storage routine's, and the in terpretation ^ of * the data is 
in preparation by Weaver and Tatsuoka for /p.utli^^-afcioh in ^ 
late T977. . - ' . 
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